USFD: a unified storage framework for SOAR HPC scientific workflows

Emerging scientific workflows in HPC focus more on analysis rather than simulation. Simulation output is so dense with information that copious amounts of analysis must be performed on a single output to understand the results of that simulation. We identify this repetitive analysis as a new application type, Simulate Once Analyze Repeatedly (SOAR) Computing. Current scientific HPC, when extended to SOAR computing, results in excessive data migration between compute and storage resources. For a workflow bound by file I/O, a large data migration overhead is unacceptable. We propose a framework which uses a data-intensive storage cluster coupled with an interoperability layer, called USFD. USFD is a Unified Storage Framework designed to better support SOAR HPC scientific workloads through enhanced file I/O support and co-located storage and analysis. In this work we analyze the performance of USFD and other traditional HPC approaches for SOAR scientific workloads. Our results show that SOAR workflows which use USFD complete analysis at a 7.5x performance increase over other approaches with QCD and 4x performance increases with FLASH.